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DETAILED ACTION 
Response to Arguments 

1 . The Request for Continued Examination filed 1 0/1 5/2007 have been entered and 
made of record. 

2. The Applicant has canceled claim(s) 7 and 9. 

3. The Applicant has included newly added claim(s) 12. 

4. The application has pending claim (s) 1-6, 8, and 10-12. 

5. Applicant's arguments with respect to claim(s) 1-6, 8, and 10-12 have been 
considered but are moot in view of the new ground(s) of rejection because of the 
Request for Continued Examination (RCE). 

6. Applicant's arguments filed 1 0/1 5/2007 have been fully considered but they are 
not persuasive. 

The Applicant alleges, "A notable feature of Claim 1 ..." in page 5, "Applicants 
respectfully submit that they understand Simpson ..." in page 6, and "For at least these 
reasons ..." in page 6, and states respectively that the prior art reference Simpson does 
not teach generating a revised image classification of each image based at least on the 
respective initial image classification as recited in claim 1 and that "none of the other 
cited references are cited by the Final Office Action as teaching or suggesting the 
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above-discussed features of Claim 1". Firstly the Examiner disagrees because this 
limitation that the revised image classification of each image is based at least on the 
respective initial image classification as recited in the amended independent claim 1 
was not part of the claim language in the Amendment After Non-Final dated 5/16/2007 
and therefore the issue was never raised by the Examiner during the Final Office Action 
dated 7/13/2007. Secondly the Examiner agrees that Simpson does not specifically 
disclose that the RNNCCS is based on the results of the FFNN and therefore the 
revised image / RNNCCS as discussed in the Final Office Action does not depend on 
the initial image classification / FFNN. However, due to further search and 
consideration, Simpson in view of Loui ("Automatic image event segmentation and 
quality screening for Albuming Applications" - IEEE - July 2000, cited by the Applicant in 
the Information Disclosure Sheet IDS filed on 11/13/2003) does teach the amended 
limitation as recited in claim 1; refer below to the rejection section for further 
discussions. Therefore, claim 1 is not in condition for allowance and is still not 
patentably distinguishable over the prior art references. Similarly the other claims that 
depend from independent claim 1 are also still not in condition for allowance. 

The Applicant alleges, "For example, Claim 5 requires that the pre-determined 
temporal context ..." in pages 6-7, and states respectively that the prior art reference 
Simpson does not suggest a temporal context model that is dependent on elapsed time 
between consecutive image in the sequence because the mere evaluation of two 
consecutive images in a sequence does not show a consideration of elapsed time 
between the two consecutive images as recited in claim 5. However the Examiner 
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disagrees because Simpson does state that the temporal context model / Recurrent 
Neural Network (RNNCCS) does depend on the elapsed time between the two 
consecutive images (see Simpson, page 2139, section 2 - Image Sequence 
Classification Using Recurrent Neural Networks, first paragraph, the Recurrent Neural 
Network uses the twelve daylight scenes per day evenly hourly (hourly or faster) spaced 
sequence of images in a time series manner for further improvement in classification, 
the Neural Network clearly is using that specific twelve scenes which are produced 
hourly per daytime elapse in time, it is not using the last scene from the current daytime 
and then the next first scene in the next daytime). Therefore, claim 5 is not in condition 
for allowance and is still not patentably distinguishable over the prior art. 

The Applicant alleges, "For another example, newly added Claim 12 ..." in page 
7, and states respectively that the prior art reference Simpson does not suggest that the 
revised image classification produces different results due to different elapsed times 
between a particular pair of consecutive images as recited in claim 12 respectively. The 
Examiner agrees that the Simpson reference does not specifically suggest that the 
revised image classification is dependent upon the initial image classification. However, 
Simpson in view of Loui does teach the amended limitation as recited in claim 12; refer 
below to the rejection section for further discussions. Therefore, claim 12 is not in 
condition for allowance and is still not patentably distinguishable over the prior art 
references. 
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Claim Rejections - 35 USC § 103 

7. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by Die manner in which the invention was made. 

8. Claims 1-2, 4-5 and 12 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Simpson ("A recurrent neural network classifier for improved 
retrievals of areal extent of snow cover" - IEEE - vol 39, Oct 2001 , pages 2135-2147, as 
applied in previous Office Action) in view of Loui ("Automatic image event segmentation 
and quality screening for Album ing Applications" - IEEE - July 2000, cited by the 
Applicant in the Information Disclosure Sheet IDS filed on 11/13/2003). 

Re Claim 1: Simpson discloses a method / single image classification using feed- 
forward neural networks (FFNN) and image sequence classification using recurrent 
neural networks (RNNCCS) (see pages 2138-2139, section B. Present Approaches, 
abstract, lines 8-10) for improving scene classification of a sequence of digital images / 
sequence of twelve daylight scenes per day (see page 2139, section 2, paragraph 1) 
comprising the steps of (a) providing a sequence of images / sequence of twelve 
daylight scenes per day captured in temporal succession / temporal sampling into time 
series (see page 2139, section 2 - Image Sequence Classification Using Recurrent 
Neural Networks, first paragraph); (b) classifying each of the images individually / Single 
Image Classification using Feed-Forward Neural Networks (FFNN) based on 
information / spectral and textural contained in the individual image / single image using 
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no feedback to generate an initial image classification / single image classification for 
each of the images / sequence of twelve daylight scenes per day (Feed Forward Neural 
Network (FFNN) or in other words the single FFNN or in other words the three layer 
Feed Forward Neural Network (NNCCS): each image is individually classified [no 
feedback], see page 2139, Fig. 3a, abstract, lines 8-10); and (c) generating the 
respective initial image classification / FFNN and a predetermined temporal context 
model / Recurrent Neural Network (RNNCCS) that considers at least the temporal 
succession / temporal sampling into time series of the sequence of images / sequence 
of twelve daylight scenes per day (Recurrent Neural Network (RNNCCS): time series 
dependent sequence classification [feedback], see page 2139, Fig. 3b, section 2, 
paragraph 1, "time series" shows that the temporal succession of the sequence is 
considered when inputting data into the Recurrent Neural Network, abstract, lines 8-1 0, 
page 2139, section 2, paragraph 2 "THE RNNCCS network lines 6-8, "more 
accurate than a single FFNN" shows a more accurate classification occurs and also that 
a comparison of the classification of the twelve daylight scene images sequence 
between the two networks [Single FFNN and Recurrent Neural Network] is considered 
and that these networks indeed do classify each image in this twelve scene per day 
sequence); and (d) storing the image classifications / FFNN and RNNCCS in a 
computer readable storage medium / RAM (see Simpson, page 2145, section - D. 
Performance Issues, the Neural Network classifications are processed in a computer 
environment and therefore are stored in the RAM or magnetic disk memory). 
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However, Simpson fails to specifically suggest that at least two pairs of 
consecutive images in the sequence of images have different elapsed times between 
their capture, generating a revised image classification for each image based at least on 
the respective initial image classification and a predetermined temporal context model, 
and storing the revised image classification. 

Loui discloses at least two pairs of consecutive images / adjacent pictures in the 
sequence of images / sequence of chronologically ordered photos having different 
elapsed times / uneven time interval or time difference between their capture (see Loui, 
Fig. 2, section - II. Image Event Segmentation, paragraph 1 , lines 12-16, the time 
interval or the time difference between adjacent pictures are uneven as seen in the 
Figure 2 where some adjacent pictures are very close in relative elapsed time whereas 
some adjacent pictures are far spread in relative elapsed time); generating a revised 
image classification / refinement (see Loui, Fig. 1) for each image based at least on the 
respective initial image classification / classification using image content and a 
predetermined temporal context model / classification using contextual image time data 
(see Loui, section - 1. Introduction, paragraph 1, lines 12-17 and 20-24, Fig. 1, Loui 
event classifies images using content image data [this is similar to Simpsons initial 
FFNN single classification using spectral and textural image content] and event 
classifies images using contextual time data [this is similar to Simpsons RNNCCS 
Recurrent Neural Network classification using the time series image sequence] and 
creates a revised / refinement event classification using both event classifications); and 
storing the revised / refinement image classification (see Loui, Figs. 4 and 5, image 
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signal processing is accomplished by computer systems and the output of Loui's event 
classification is seen in the computer type output in figures 4 and 5 [ this is similar to 
Simpsons image signal processing classifications being stored in the RAM or magnetic 
disk memory of the computer system]). 

Therefore, it would be obvious to one of ordinary skill in the art at the time the 
invention was made to modify Simpson's method using Loui's teachings by including to 
Simpsons hourly or faster temporal sampling the ability to have consecutive images with 
different elapsed times in order to better relate adjacent pictures with clustering (see 
Loui, Fig. 2, Section - II. Image Event Segmentation, paragraph 1, lines 1-4) and by 
including the generation of the revised / refined image classification after Simpsons 
FFNN [initial image classification] and RNNCCS [Recurrent Neural Network] 
classifications in order to combine the content image data and contextual image time 
data into the refined event classification (see Loui, Fig. 1, Section - 1. Introduction, 
paragraph 1 , lines 20-24). 

Re Claim 2: Simpson further discloses the information used in step (b) includes pixel 
information / spectral and textural (see page 2138, section 1 - Single Image 
Classification Using Feed-Forward Neural Networks, first paragraph). 

Re Claim 4: Loui further discloses the pre-determined temporal context model in step 
(c) is independent of elapsed time between consecutive images (see Loui, Section - II. 
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Image Event Segmentation, paragraph 1, lines 8-10, there could be an instance when 
no time interval data is available). 

Re Claim 5: Simpson further discloses the pre-determined temporal context model in 
step (c) is dependent on elapsed time / temporal sampling into time series between 
consecutive images (see Simpson, page 2139, section 2 - Image Sequence 
Classification Using Recurrent Neural Networks, first paragraph, the Recurrent Neural 
Network uses the twelve daylight scenes per day evenly hourly (hourly or faster) spaced 
sequence of images in a time series manner for further improvement in classification, 
the Neural Network clearly is using that specific twelve scenes which are produced 
hourly per daytime elapse in time, it is not using the last scene from the current daytime 
and then the next first scene in the next daytime). 

Re Claim 12: Loui further discloses wherein the predetermined temporal context model 
in step (c) is dependent on elapsed time between consecutive images in the sequence, 
such that different elapsed times between a particular pair of consecutive images 
produces a different revised image classification for a later-captured image of the 
particular pair of consecutive images (see Loui, Fig. 2, section - II. Image Event 
Segmentation, paragraph 1, lines 12-16, the time interval or the time difference between 
adjacent pictures are uneven as seen in the Figure 2 where some adjacent pictures are 
very close in relative elapsed time whereas some adjacent pictures are far spread in 
relative elapsed time, the contextual image time data is used along with the content 
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image data for the refined classification, Figs. 4 and 5 show that different elapsed times 
between particular pair of consecutive images produce different event classification 
results, if a pair of images are real close in elapsed time they will most likely [also 
dependent upon image content data] be classified in the same event, if a pair of images 
are far in elapsed time they will most likely [also dependent upon image content data] 
be classified in different events). 

9. Claim 3 is rejected under 35 U.S.C. 103(a) as being unpatentable over Simpson 
as modified by Loui, and in further view of Tretter et al (US 6,977,679 B2, as applied in 
previous Office Action). The teachings of Simpson as modified by Loui have been 
discussed above. 

However, Simpson as modified by Loui fails to specifically suggest the 
information used in step (b) includes capture-device generated metadata information. 

Tretter, as recited in claim 3 . discloses the information used in step (b) includes 
capture device generated / digital camera metadata / focusing distance information (see 
abstract, lines 6-1 3). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to further modify Simpson's method, as modified by Loui, 
using Tretter's teachings by including focusing distance metadata to Simpson's step (b) 
in order to enhance the classification of snow cover by further distinguishing clouds and 
snow cover by identifying the difference in distance between clouds and the snow cover 
from the satellite which captures images. 
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10. Claims 6, 8, and 10-11 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Simpson as modified by Loui, and further in view of Huang 
("Integration of multimodal features for video scene classification based on HMM" - 
IEEE - Sept 1999, pages 53-58, as applied in previous Office Action). The teachings of 
Simpson as modified by Loui have been discussed above. 

However, Simpson as modified by Loui fails to specifically suggest that the 
temporal context model is a non-casual model dependent on both a previous image and 
a subsequent image. 

Huang, as recited in claim 8 . discloses the pre-determined temporal context 
model is a non-casual model / discrete ergodic Hidden Markov Model dependent on 
both the previous image and a subsequent image / visited from any state (see page 55, 
section Product HMM, page 56, section SIMULATION RESULTS, second paragraph, 
the discrete ergodic HMM visits any states or images from any state or image which 
makes the model non-causal). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to further modify Simpson's method, as modified by Loui, 
using Huang's teachings by replacing a non-causal discrete ergodic Hidden Markov 
Model with Simpson's temporal context model in order to give more correlation data 
between all the images of the sequence instead of just a few images (images before the 
current image) to further improve the accuracy of the classification. 
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Although Simpson's temporal context model, as modified by Loui, and in further 
view of Huang's modifications teaches a non-causal discrete ergodic Hidden Markov 
Model, it does not specifically disclose, as recited in claim 6 , that the temporal context 
model is the causal Hidden Markov Model dependent on a previous image, and it does 
not specifically disclose, as recited in claim 10 . that the temporal context model is 
imposed using Viterbi algorithm, and it does not specifically disclose, as recited in claim 
11, that the temporal context model is imposed using a belief propagation algorithm. It 
would have been obvious to one of ordinary skill in the art at the time the invention was 
made though to have such a feature of causality in a HMM model, a Viterbi algorithm, or 
a belief propagation algorithm for a temporal context model because they are just other 
methods of computing the probability for classification of a particular sequence which 
Huang's non-causal Hidden Markov Model is basically accomplishing. 

Conclusion 

1 1 . Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Bernard Krasnic whose telephone number is (571 ) 270- 
1357. The examiner can normally be reached on Mon-Thur 8:00am-4:00pm and every 
other Friday 8:00am-3:00pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Jingge Wu can be reached on (571 ) 272-7429. The fax phone number for 
the organization where this application or proceeding is assigned is 571 -273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Bernard Krasnic 
December 1 1 , 2007 



